1. Technical Field
A “Scene Re-Lighter” provides various techniques for relighting sparsely captured images, and in particular, various techniques for using an automatically reconstructed light transport matrix to provide various combinations of complex light transport effects in images, including caustics, complex occlusions, inter-reflections, subsurface scattering, etc.
2. Related Art
Image-based relighting offers an unparalleled advantage of realistic rendering without scene modeling, which is often an arduous task. However, to generate high quality results, conventional relighting techniques typically require tens of thousands of images for accurate reconstruction of a “light transport matrix” (i.e., matrix T). Several conventional approaches have attempted to reduce the number of required images. Unfortunately, these techniques either are dedicated to specific light transport effects, or are primarily effective with scenes of simple geometry configurations. Consequently, applying such techniques to scenes with complex lighting effects, occlusions, and/or complex geometries still requires a large number of input images and computationally expensive light transport matrix reconstruction methods.
As is well known to those skilled in the art, the idea behind image-based relighting is to directly capture the “light transport” of a real-world scene so that it can be rendered with new illumination. Mathematically, image-based relighting can be formulated as the following equation:b=T·l  Equation (1)where T is an m×n light transport matrix that describes the light transport from n light sources to m camera pixels, l is the illumination condition represented by a vector of incident radiance from n light sources, and b is the outgoing radiance observed in a camera image with m pixels. Hence, image-based relighting often focuses on constructing or recovering the matrix T, which can then be used to provide a variety of lighting effects.
In general, the light transport matrix T represents discrete samples of the reflectance field. Conventionally, a complete 8D reflectance field, which describes the light transport from the incident light field to the outgoing light field, is difficult to capture and process. Therefore, most existing methods consider simplified 4D and 6D reflectance fields instead of the more complex 8D reflectance field.
Conventional light transport acquisition methods can be generally categorized into one of three basic classes: brute force, sparsity based, and coherence based methods.
The brute force methods capture very large numbers of images to directly measure the light transport matrix from the scene, where each column is an image of the scene lit by a single light source in the incident light domain. One such technique uses a “light stage device” for capturing 4D reflectance fields for a fixed viewpoint and distant lighting by moving a point light source around the object. A related technique exploits the well-known “Helmholtz reciprocity” to capture the reflectance field of highly reflective objects. To obtain dense samples in the incident light domain, rows of the light transport matrix are captured by shooting rays from the viewpoint and capturing high-resolution images of the scene projected over the incident light domain. Reciprocity is also exploited to acquire 6D reflectance fields. Unfortunately, all of these methods require tens of thousands of images for modeling a high quality light transport matrix.
The sparsity based methods model the light transport matrix with a set of basis functions defined over the incident light domain and assume that each row of the light transport matrix can be approximated by a linear combination of a sparse set of basis functions. Thus, the light transport matrix can be reconstructed by deriving the sparse basis and their weights for each row from a set of images captured under specific lighting conditions. “Environment matting” models the reflectance of specular or refractive objects by representing the light transport of each pixel (i.e., a row of transport matrix) with a single 2D box function.
Such techniques have been extended for modeling glossy objects by replacing the box function with an oriented Gaussian kernel. A related technique models the light transport matrix with hierarchical rectangular basis functions. Further, an adaptive algorithm is used to derive the sparse basis and their weights for each pixel from images of the scene captured under various natural illumination conditions. Another of these techniques operates by modeling the light transport matrix with wavelets and inferring the light transport matrix from images of the scene illuminated by carefully designed wavelet noise patterns. Both approaches apply a greedy strategy to find a suboptimal sparse basis for each pixel, which only works well for scenes with simple occlusions. Yet another technique uses a compressive sensing approach that computes the solution for each pixel from images captured from a scene illuminated by patterned lighting. However, the number of images needed for reconstruction depends on both the row length and the number of basis used for each row, which becomes quite large for scenes with complex occlusions. The reconstruction process is also computationally expensive and time consuming.
Coherence based methods acquire the light transport matrix by exploiting the coherence of reflectance field data. One such technique is based on the interpolation and compression of the reflectance field. Another coherence based technique uses an adaptive sampling scheme for sampling the 4D reflectance field. The spatial coherence of the reflectance field in the incident domain is exploited for accelerating the acquisition process. Starting from a set of images taken with a sparse set of regularly distributed lighting directions, the algorithm analyzes observed data and then captures more images in each iteration with the new lighting directions where the reflectance field is not smooth. Note that the smoothness of reflectance data among neighboring pixels is also exploited in various coherence-based techniques for improving the result quality.
In addition to the reconstruction of low rank symmetric matrices, the Nyström method is also widely used in the machine learning community for approximately computing the eigenvalues and eigenvectors of a symmetric matrix from sparse matrix samples. For example, one conventional Nyström-based technique addresses asymmetric matrices by using a “pseudo-skeleton approximation” for reconstructing a complete matrix from a sparse collection of its rows and columns. In graphics research, another Nyström-based technique is used to accelerate appearance edit propagation by approximating the dense symmetric distance matrix with sparsely sampled rows and columns.
Similar techniques have been used to efficiently render synthetic scenes. In this approach, columns of the matrix are clustered in a small number of groups according to their values in the sparsely sampled rows. A representative column for each group is then sampled and weighted for approximating other columns in the same group. However, this approach only uses the coherence between the columns for approximating the matrix. Coherence between the rows is not exploited. Moreover, applying this approach to the light transport matrix for use in relighting applications has been observed to generate temporal artifacts under animated light.